A New Pyramid Algorithm For Video Compression And Video Analysis

ABSTRACT

A method of representing image data comprising a spatial pyramid data structure of order L, said method comprising: providing an initial representation of said image data, said representation comprising a set of lattices of level 0; connecting each said lattices of level 0 to said image data; performing L reduction steps each comprising: for each one of at least one current lattice of the current level, optionally blurring said current lattice; providing a plurality of down sampled sub-lattices of said current optionally blurred lattice, said at least one current lattice being the parent lattice of said respective plurality of down sampled sub-lattices; and conecting said plurality of down sampled sub-lattices to said respective parent lattice, wherein said plurality of down sampled sub-lattices of the current level, comprise the set of current lattices for the next level.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This patent application claims priority from and is related to U.S.Provisional Patent Application Ser. No. 62/336,770, filed on 16 May2016, this U.S. Provisional Patent Application incorporated by referencein its entirety herein.

TECHNOLOGY FIELD

The present invention is in the field of video compression and videoanalysis.

BACKGROUND

A video consists of a sequence of frames, with a given spatial andtemporal resolution. The spatial resolution is given by the number ofpixels sampled along the x and y space directions, so called the numberof columns and rows of the frame. The temporal resolution is given bythe number of pixels sampled along the time direction so called thenumber of frames in the video. We can therefore view the sampled frameas a two dimensional rectangular lattice with corresponding sub-latticesas in FIG. 1, In diagram 110, we display the four Dyadic sub-lattices ofthe frame lattice, namely: the Dyadic-Even-Even sub-lattice of theTriangle pixels, the Dyadic-Even-Odd sub-lattice of the Diamond pixels,the Dyadic-Odd-Even sub-lattice of the Square pixels, and theDyadic-Odd-Odd sub-lattice of the Star pixels, In diagram 120 we displaythe two Quincunx sub-lattices of the frame lattice, namely: theQuincunx-Even sub-lattice of the Black pixels and the Quincunx-Oddsub-lattice of the White pixels. Note that whereas the Dyadicsub-lattices are regular rectangular frame lattices as the originalframe lattice is, the Quincunx sub-lattices are not. In order to makethem regular, we can rotate them for example by 45° clockwise. In FIG.2, we display the Quincunx-Even sub-lattice. Note that the un-rotatedQuincunx-Even sub-lattice frame in diagram 210 is non rectangular. Therotated Quincunx-Even sub-lattice of diagram 220 is rectangular, butstill not regular as the original frame lattice is because of themissing pixels on the four corners of the frame. In FIG. 3 we displaythe two Quincunx-Even's sub-lattices. As displayed in diagram 310, theQuincunx-Even-Even sub-lattice is denoted by 1, and theQuincunx-Even-Odd sub-lattice is denoted by 2. These Quincunx-Even'ssub-lattices, as displayed in diagram 320, are again not regular framesbecause of the missing pixels at the four corners of the frame. In orderto make them regular we need to un-rotate them counter clockwise by 45°.The resulting un-rotated Quincunx-Even's sub-lattices, displayed indiagram 330, now become regular rectangular frame lattices as theoriginal frame lattice is. Note further that the Quincunx-Even-Evensub-lattice corresponds to the Dyadic-Even-Even sub-lattice, and thatthe Quincunx-Even-Odd sub-lattice corresponds to the Dyadic-Even-Oddsub-lattice. In FIG. 4, we display similarly the Quincunx-Oddsub-lattice. Note again that the un-rotated Quincunx-Odd sub-latticeframe in diagram 410 is non rectangular. The rotated Quincunx-Oddsub-lattice of diagram 420 is rectangular, but still not regular as theoriginal frame lattice because of the missing pixels on the four cornersof the frame. In FIG. 5, we display the two Quincunx-Odd's sub-lattices.As displayed in diagram 510, the Quincunx-Odd-Even sub-lattice isdenoted by 3, and the Quincunx-Odd-Odd sub-lattice is denoted by 4.These Quincunx-Odd's sub-lattices, as displayed in diagram 520, areagain not regular frames because of the missing pixels at the fourcorners of the frame. In order to make them regular we need to un-rotatethem counter clockwise by 45°. The resulting un-rotated Quincunx-Odd'ssub-lattices, displayed in diagram 530, now become regular framelattices as the original frame lattice is. Here the Quincunx-Odd-Evensub-lattice corresponds to the Dyadic-Odd-Even sub-lattice, and theQuincunx-Odd-Odd sub-lattice corresponds to the Dyadic-Odd-Oddsub-lattice.

We therefore conclude that any two consecutive Quincunx down samplingoperations correspond to one Dyadic down sampling operation, resultingin a regular frame lattice as the original frame lattice is.

SUMMARY

BRIEF DESCRIPTION OF THE DRAWINGS

For better understanding of the invention and to show how the same maybe carried into effect, reference will now be made, purely by way ofexample, to the accompanying drawings.

With specific reference now to the drawings in detail, it is stressedthat the particulars shown are by way of example and for purposes ofillustrative discussion of the preferred embodiments of the presentinvention only, and are presented in the cause of providing what isbelieved to be the most useful and readily understood description of theprinciples and conceptual aspects of the invention. In this regard, noattempt is made to show structural details of the invention in moredetail than is necessary for a fundamental understanding of theinvention, the description taken with the drawings making apparent tothose skilled in the art how the several forms of the invention may beembodied in practice. In the accompanying drawings:

FIG. 1 is a diagram of a frame lattice and some of its sub-lattices;

FIG. 2 is a diagram of the Quincunx-Even sub-lattice;

FIG. 3 is a diagram of the Quincunx-Even's sub-lattices;

FIG. 4 is a diagram of the Quincunx-Odd sub-lattice;

FIG. 5 is a diagram of the Quincunx-Odds sub-lattices;

FIG. 6 is a diagram of some additional interpolated frames;

FIG. 7 is a diagram of an example of a complete Spatial Pyramid withL=2, N=2, and q=½;

FIG. 7a is a diagram of an example of a partial Spatial Pyramid withL=2, N=2, and q=½;

FIG. 8 is a diagram of an example of a complete Quincunx Spatial Pyramidwith L=2, and q=½;

FIG. 8a is a diagram of an example of a partial Quincunx Spatial Pyramidwith L=2, and q=½;

FIG. 9 is a diagram of an example of optical flow at odd levelsk=(2m+1),

FIG. 10 is a diagram of an example of the up scaled optical flow of FIG.9; and

FIG. 11 is a diagram of an example of spatial pixel prediction.

DETAILED DESCRIPTION OF EMBODIMENTS

In the present invention we present a new pyramid data structure andalgorithms for video compression and video analysis.

The Spatial Pyramid of Sub-Lattices

A Spatial Pyramid is created using the current frame lattice and itsrelated sub-lattices.

Initialization

We begin by creating additional frames of the same resolution as theoriginal frame using interpolation. For example, we may interpolate theoriginal frame by half pixel along the x (columns) direction, or by halfpixel along the y (rows) direction or by half pixel along both the x andy directions. In FIG. 6, we show such possible half pixelinterpolations. We assume here that the original frame, displayed indiagram 610, corresponds to the Dyadic down-sampled Triangle sub-latticeof the frame displayed in diagram 110 of FIG. 1. Then, half pixelinterpolations of the original frame along the x direction, as displayedin diagram 620, correspond to the Dyadic down-sampled Square sub-latticeof the frame in diagram 110. Similarly, half pixel interpolations alongthe y direction as displayed in diagram 630, correspond to the Dyadicdown-sampled Star sub-lattice of the frame in diagram 110, and halfpixel interpolations along both the x and y directions, as displayed indiagram 640, correspond to the Dyadic down-sampled Diamond sub-latticeof the frame in diagram 110. We can similarly create additional framesby quarter pixel interpolation etc.

In general, let

$q = \frac{1}{p}$

denotes the q-fractional interpolation method used to create theadditional frames, and let Z denotes the original frame. Then we beginwith a set of p² shifted frames of the same resolution as the originalframe, which we denote as:

Z_(i,j), 0≤i, j<p.

Here the i-index above represents the q-fractional interpolation of theoriginal frame along the y-direction, and the j-index above representssimilarly the q-fractional interpolation of the original frame along thex-direction. Hence, for example, frame Z_(0,0) corresponds to theoriginal frame Z, frame Z_(1,0) corresponds to the frame resulting from

$q = \frac{1}{p}$

fractional interpolation of the original frame Z along the y-direction,etc. We now define the Spatial Pyramid for this set of p² shifted framesas defined above. Let L≥0 be some integer defining the order of theSpatial Pyramid. Then we begin by denoting the 0-level of the pyramid bythe set of lattices

Z_(i,j) ^(0,0)≡Z_(i,j), 0≤i,j<p,

where the first top index represents the level number, and the secondtop index represents a global numbering for that lattice. We thenconnect each such lattice to the original frame, see for example FIG. 7,and FIG. 7 a.

Now, assume we are using some down sampling method, such as Quincunxdown sampling, Dyadic down sampling etc. Then, let N denote the numberof sub-lattices for that method. For example, for the Quincunx downsampling method N=2, and for the Dyadic down sampling method N=4.

We now proceed to define the creation of the Spatial Pyramid of order L.

Step k=0, . . . , L−1

We first optionally blur each frame lattice of level k to get theblurred frame lattices:

$\left. Z_{i,j}^{k,n_{k}}\rightarrow{\overset{\sim}{Z}}_{i,j}^{k,n_{k}} \right.,\left\{ \begin{matrix}{0n_{k} < N^{k}} \\{{0i},{j < p}}\end{matrix} \right.$

Next, given said above down sampling method, with N possiblesub-lattices, we create all the possible sub-lattices for each of theseoptionally blurred frame lattices, namely:

$\left. {\overset{\sim}{Z}}_{i,j}^{k,n_{k}}\rightarrow Z_{i,j}^{{k + 1},{{n_{k}N} + n}} \right.,\left\{ \begin{matrix}{{n = 0},\ldots \;,{N - 1}} \\{0n_{k} < N^{k}} \\{{0i},{j < p}}\end{matrix} \right.$

Finally, we denote all these down-sampled sub-lattices as:

$Z_{i,j}^{{k + 1},n_{k + 1}},\left\{ \begin{matrix}{0n_{k + 1} < N^{k + 1}} \\{{0i},{j < p}}\end{matrix} \right.$

and call them the respective frame lattices of level (k+1).

The Pyramid Relations

We construct the Pyramid by connecting the frame lattices of each level,namely

$Z_{i,j}^{k,n_{k}},\left\{ \begin{matrix}{0 \leq k < L} \\{0n_{k} < N^{k}} \\{{0i},{j < p}}\end{matrix} \right.$

to their respective frame lattices of the next level, namely:

$Z_{i,j}^{{k + 1},{{n_{k}N} + n}},{n = 0},\ldots \;,{N - 1},\left\{ {\begin{matrix}{0 \leq k < L} \\{0n_{k} < N^{k}} \\{{0i},{j < p}}\end{matrix}.} \right.$

In this context we define {tilde over (Z)}_(i,j) ^(k,n) ^(k) to be theparent of Z_(i,j) ^(k+1,n) ^(k) ^(N+n) and we denote this as

${{\overset{\sim}{Z}}_{i,j}^{k,n_{k}} = {{Par}\left( Z_{i,j}^{{k + 1},{{n_{k}N} + n}} \right)}},{n = 0},\ldots \;,{N - 1},\left\{ \begin{matrix}{0 \leq k < L} \\{0n_{k} < N^{k}} \\{{0i},{j < p}}\end{matrix} \right.$

An example of a complete Spatial Pyramid and its pyramid relations forL=2, N=2 and q=½ is depicted in FIG. 7. In practice, we can apply thereduction step to only some of the respective lattices of thecorresponding level, yielding a partial Spatial Pyramid as for exampledepicted in FIG. 7a . for L=2, N=2, and q=½.

EXAMPLE The Quincunx Spatial Pyramid

In this section we give a more concrete example of the Quincunx SpatialPyramid.

Initialization

We begin by creating additional frames of the same resolution as theoriginal frame using interpolation as was done in the previous section.Here, we let q=2^(−p) denote the fractional interpolation method used tocreate the additional frames, so that the total number of initial framesis now 4^(p), namely:

Z_(i,j), 0≤i, j<2^(p).

Here the index above represents again the q-fractional interpolation ofthe original frame along the y-direction, and the j-index aboverepresents similarly the q-fractional interpolation of the originalframe along the x-direction. Hence, for example, frame Z_(0,0)corresponds to the original frame Z, frame Z_(1,0) corresponds to theframe resulting from q=2^(−p) fractional interpolation of the originalframe Z along the y-direction, etc. We now use the Quincunx downsampling method to create the Quincunx Spatial Pyramid of order L usingthe following notations for level 0:

Z_(i,j) ⁰≡Z_(i,j), 0≤i,j<2^(p),

In what follows we assume the distances between the pixels of theoriginal frame, along the x and the y directions, are the same, and areequal to 1. We now distinguish between Even and Odd steps as follows.

Even Step k=0, 2, 4, . . .

Let the frame lattices of level k be denoted by

$Z_{i,j}^{k},\left\{ \begin{matrix}{{0i},{j < 2^{m + p}}} \\{k = {2m}}\end{matrix} \right.$

Here, we can assume by induction that the pixel distances along the xand the y directions are (√{square root over (2)})^(k)=2^(m). Note alsothat the ith and jth indices of each such lattice correspond to therespective fractional shift of the top left pixel of that lattice withrespect to the top left pixel of the original lattice.

Then, as in the previous section, we first optionally blur each suchframe lattice to get the optionally blurred frame lattices:

$\left. Z_{i,j}^{k}\rightarrow{\overset{\sim}{Z}}_{i,j}^{k} \right.,\left\{ \begin{matrix}{{0i},{j < 2^{m + p}}} \\{k = {2m}}\end{matrix} \right.$

Then, we create for each of these optionally blurred lattices, theQuincunx-Even sub-lattice and the Quincunx-Odd sub-lattice which wedenote as:

$\left. {\overset{\sim}{Z}}_{i,j}^{k}\rightarrow Z_{i,i}^{k + 1} \right.,Z_{i,{i + 2^{m + p}}}^{k + 1},\left\{ \begin{matrix}{{0i},{j < 2^{m + p}}} \\{k = {2m}}\end{matrix} \right.$

Or more simply as:

$Z_{i,j}^{k + 1},\left\{ \begin{matrix}{0i < 2^{m + p}} \\{0j < 2^{m + p + 1}} \\{k = {2m}}\end{matrix} \right.$

Note that the ith and jth indices of each of these un-rotated Quincunxsub-lattices, as depicted in diagram 210 and diagram 410 respectively,correspond again to the fractional-shift of the top left pixels withrespect to the top left pixel of the original frame. Furthermore, thepixel distances along the x and the y directions of these rotatedQuincunx sub-lattices, as depicted in diagram 220 and diagram 420respectively, are now √{square root over (2)}^((k+1))=√{square root over(2)}*2^(m). In this context we define similarly {tilde over (Z)}_(i,j)^(k) to be the parent of these corresponding sub-lattices, and we denotethis as:

${{\overset{\sim}{Z}}_{i,j}^{k} = \begin{matrix}{{Par}\left( Z_{i,j}^{k + 1} \right)} \\{{Par}\left( Z_{i,{j + 2^{m + p}}}^{k + 1} \right)}\end{matrix}},\left\{ \begin{matrix}{{0i},{j < 2^{m + p}}} \\{k = {2m}}\end{matrix} \right.$

Step k=1, 3, 5, . . .

Let the frame lattices of level k be denoted as above, i.e.

$Z_{i,j}^{k},\left\{ \begin{matrix}{0i < 2^{m + p}} \\{0j < 2^{m + p + 1}} \\{k = {{2m} + 1}}\end{matrix} \right.$

Then, as observed above, the pixel distances along the x and the ydirections of these rotated Quincunx sub-lattices, as depicted indiagram 220 and diagram 420 respectively, are now √{square root over(2)}^(k)=√{square root over (2)}*2^(m). Similarly, the ith and jthindices of each of these un-rotated Quincunx sub-lattices, as depictedin diagram 210 and diagram 410 respectively, correspond to thefractional-shift of the top left pixels with respect to the top leftpixel of the original frame.

Then, as in the previous section, we first optionally blur each suchframe lattice to get the optionally blurred frame lattices:

$\left. Z_{i,i}^{k}\rightarrow{\overset{\sim}{Z}}_{i,j}^{k} \right.,\left\{ \begin{matrix}{0i < 2^{m + p}} \\{0j < 2^{m + p + 1}} \\{k = {{2m} + 1}}\end{matrix} \right.$

Then, we create for each of these optionally blurred lattices, theQuincunx-Even sub-lattice and the Quincunx-Odd sub-lattice which wedenote as:

$\left. {\overset{\sim}{Z}}_{i,j}^{k}\rightarrow Z_{i,j}^{k + 1} \right.,Z_{{i + 2^{m + p}},{{({j + 2^{m + p}})}{{mod}{(2^{m + p + 1})}}}}^{k + 1},\left\{ \begin{matrix}{0i < 2^{m + p}} \\{0j < 2^{m + p + 1}} \\{k = {{2m} + 1}}\end{matrix} \right.$

Or more simply as

$Z_{i,j}^{k + 1},\left\{ \begin{matrix}{{0i},{j < 2^{m + p + 1}}} \\{k = {{2m} + 1}}\end{matrix} \right.$

Note that by now, these sub-lattices become regular rectangularlattices. The ith and jth indices of each of these lattices correspondagain to the fractional-shift of the top left pixels with respect to thetop left pixel of the original frame. Furthermore, the pixel distancesalong the x and the y directions are now √{square root over(2)}^((k+1))=2^(m+1). In this context we define similarly {tilde over(Z)}_(i,j) ^(k) to be the parent of these corresponding sub-lattices,and we denote this as:

${{\overset{\sim}{Z}}_{i,j}^{k} = \begin{matrix}{{Par}\left( Z_{i,j}^{k + 1} \right)} \\{{Par}\left( Z_{i,{j + 2^{m + p}},{{({j + 2^{m + p}})}{mo}\; {d{(2^{m + p + 1})}}}}^{k + 1} \right)}\end{matrix}},\begin{matrix}{0i < 2^{m + p}} \\{0j < 2^{m + p + 1}} \\{k = {{2m} + 1}}\end{matrix}$

To summarize, we can denote the frame lattices of each level as follows:

Step k=0, 1, . . . , 1

$Z_{i,j}^{k} = \left\{ \begin{matrix}{{0j < 2^{k_{1} + p}},} & {k_{1} = \left\lfloor \frac{k}{2} \right\rfloor} \\{{0j < 2^{k_{2} + p}},} & {k_{2} = \left\lceil \frac{k}{2} \right\rceil}\end{matrix} \right.$

An example of a complete Quincunx Spatial Pyramid and its pyramidrelations for L=2 and q=½ is depicted in FIG. 8. In practice, we canapply the reduction step to only some of the respective lattices of thecorresponding level, yielding a partial Quincunx Pyramid as for exampledepicted in FIG. 8a . for L=2, and q=½.

An Application to Video Compression

In this section we describe the usage of the Spatial Pyramid in thecontext of video compression as discussed in the No Latency algorithmpresented in Pat [1]. The Spatial Pyramid can also be used in thecontext of many other video compression algorithms such as the onespresented in Pat [2], and Pat [3], as well as with standard videocompression algorithms such as MPEG, see Ref[1].

Introduction

We consider here the problem of encoding and decoding a sequence ofvideo frames, in the order of their appearance. In particular, we do notconsider here the problem of encoding and decoding the first frame inthe sequence, but only any of the following frames in the sequence. Wecall the frame to be encoded and decoded the current frame, and assumethat all the previously processed frames are known in their decoded formto both Encoder and Decoder. We now describe the process of encoding anddecoding the current frame using the previously decoded frames. Forsimplicity, we use here only the last previously decoded frame, althoughmore of the previously decoded frames can be used. Furthermore, for thepurpose of presentation, we use here the Quincunx Spatial Pyramid andthe Quincunx down sampling method.

Initialization

Let Y be the current frame to be encoded and decoded, let Z be the lastpreviously decoded frame, and let L denotes the order of the QuincunxSpatial Pyramid to be used. Then we begin by creating a simple pyramiddecomposition for Y≡Y⁰ as follows: Iteratively perform the followingstep for k=0, 1, . . . , L−1:

Optionally blur the frame lattice Y^(k) and denote the optionallyblurred lattice by {tilde over (Y)}^(k). Then either Quincunx-Even downsample or Quincunx-Odd down sample this optionally blurred lattice anddenote the resulting down sampled frame lattice as Y^(k+1). We finallydenote this whole resulting simple pyramid decomposition of Y by:

Y^(k), {tilde over (Y)}^(k) 0≤k<L, Y^(L).

We now create a corresponding Quincunx Spatial Pyramid of order L forthe last previously decoded frame Z, as discussed in the previoussection, in both the Encoder and the Decoder. To recap, given an initialparameter p≥0, we create the respective 4^(p) interpolated frames:

Z_(i,j) ⁰, 0≤i,j<2^(p).

Then, we create the corresponding optionally blurred lattices:

${\overset{\sim}{Z}}_{i,j}^{k},{0k < {L\left\{ \begin{matrix}{{0i < 2^{k_{1} + p}},} & {k_{1} = \left\lfloor \frac{k}{2} \right\rfloor} \\{{01 < 2^{k_{2} + p}},} & {k_{2} = \left\lceil \frac{k}{2} \right\rceil}\end{matrix} \right.}}$

where each such lattice of level k is optionally blurred in the same wayas {tilde over (Y)}^(k) was blurred. Finally we down sample each ofthese lattices, creating both the Quincunx-Even and Quincunx-Oddsub-lattices, which we then denote as:

$Z_{i,j}^{k},{0 < k \leq {L\left\{ {\begin{matrix}{{0i < 2^{k_{1} + p}},} & {k_{1} = \left\lfloor \frac{k}{2} \right\rfloor} \\{{01 < 2^{k_{2} + p}},} & {k_{2} = \left\lceil \frac{k}{2} \right\rceil}\end{matrix}.} \right.}}$

The Algorithm

We encode and decode the current frame Y using the simple pyramiddecomposition of Y and the respective Quincunx Spatial Pyramid of thelast previously decoded frame Z as discussed above. Denoting by Ŷ theresulting decoded current frame, we will obtain in the process acorresponding simple pyramid decomposition for Ŷ, i.e.

Ŷ^(k), {tilde over (Ŷ)}^(k) 0≤k<L, Ŷ^(L).

More precisely, we process the pyramids bottom up, in (L+1) steps,starting from the lowest level L of the pyramid up to the top most level0 of the pyramid. On each such step we compress/decompress therespective pyramid lattice of Y of that level, yielding the respectivepyramid lattice of the decoded frame it of that level. Therefore, by theend of the algorithm we obtain the required decoded frame it asproclaimed.

Step L: We Compress/Decompress Y^(L).

-   -   1) In the Encoder, we compress Y^(L) either directly using some        known compression method such as Jpeg or Jpeg2000, or we        compress Y^(L) indirectly as follows: We inspect the lowest        level lattices of the respective Z Quincunx Spatial Pyramid:

$Z_{i,j}^{L},\left\{ \begin{matrix}{{0i < 2^{L_{1} + p}},} & {L_{1} = \left\lfloor \frac{L}{2} \right\rfloor} \\{{0j < 2^{L_{2} + p}},} & {L_{2} = \left\lceil \frac{L}{2} \right\rceil}\end{matrix} \right.$

-   -   -   and choose the one, say Z_(i) ₀ _(,j) ₀ ^(L), which best            approximates Y^(L). Then we compress the difference between            that lattice and Y^(L), namely D^(L)=Y^(L)−Z_(i) ₀ _(,j) ₀            ^(L), using any known compression method such as for example            wavelet based compression methods, see for example Ref[3].            The Encoder then sends the respective compressed bit stream            to the Decoder. In the first case above (direct compression            of Y^(L)), the bit stream consists of the compressed lattice            Ŷ^(L). In the second case above (compression of the            difference between that lattice and Y^(L)), the bit stream            consists of the compressed lattices' difference {circumflex            over (D)}^(L) together with the indices i₀, j₀.

    -   2) In the Decoder, as well as in the Encoder, we now reconstruct        the compressed lattice Ŷ^(L). In the first case above we        decompress the bit stream to obtain Ŷ^(L) directly. In the        second case above we decompress the bit stream to obtain the        compressed lattices' difference {circumflex over (D)}^(L), and        then use the transmitted indices i₀, j₀ to obtain Ŷ^(L) by        computing Ŷ^(L)={circumflex over (D)}^(L)+Z_(i) ₀ _(,j) ₀ ^(L).

Step k=L−1, . . . , 0: We may now assume by induction, that we havealready processed all the Y pyramid lattices of level m>k, namely:

Y ^(L) , Y ^(m) , {tilde over (Y)} ^(m) m=L−1, . . . , k+1.

and obtained the respective pyramid lattices of Ŷ, namely:

Ŷ ^(L) , Ŷ ^(m) , {tilde over (Ŷ)} ^(m) m=L−1, . . . , k+1.

We now show how to proceed and obtain Ŷ^(k) and {tilde over (Ŷ)}^(k)from that respect pyramid of lattices of Ŷ and Y^(k) and {tilde over(Y)}^(k). Hence, by the end of the algorithm we obtain the decoded frameŶ≡Ŷ⁰ as required.

-   -   1) In both the Encoder and Decoder, we first compute the optical        flows, see Ref[2], between the lattice Ŷ^(k+1) and each of the        respective lattices of level (k+1) of the Z Quincunx Spatial        Pyramid:

${F_{i,j}^{k + 1} = {{OpticalFlow}\left( {\hat{Y}}^{k + 1}\rightarrow Z_{i,j}^{k + 1} \right)}},\left\{ \begin{matrix}{{0i < 2^{k_{1} + p}},} & {k_{1} = \left\lfloor \frac{k + 1}{2} \right\rfloor} \\{{0j < 2^{k_{2} + p}},} & {k_{2} = \left\lceil \frac{k + 1}{2} \right\rceil}\end{matrix} \right.$

-   -   2) Let Y ^(k) denote some proposed lattice with the same        resolution as Y^(k) and for which the lattice Ŷ^(k+1) is its        respective sub-lattice. Note that about half of the pixels in Y        ^(k) are yet unknown but they will be defined later on.    -   3) In both the Encoder and Decoder, we then upscale the optical        flows constructed in 1) above, to respective proposed optical        flows between the proposed lattice Y ^(k) and each of the        respective optionally blurred lattices of the Z Quincunx Spatial        Pyramid of that level, namely,

${{\overset{\_}{F}}_{i,j}^{k} = {{OpticalFlow}\left( {\overset{\_}{Y}}^{k + 1}\rightarrow{\overset{\sim}{Z}}_{i,j}^{k} \right)}},\left\{ \begin{matrix}{{0i < 2^{k_{1} + p}},} & {k_{1} = \left\lfloor \frac{k}{2} \right\rfloor} \\{{0j < 2^{k_{2} + p}},} & {k_{2} = \left\lceil \frac{k}{2} \right\rceil}\end{matrix} \right.$

-   -   -   This can be done for example as follows: Suppose k=2m is an            even level, and the optical flow between the lattice Ŷ^(k+1)            and the lattice Z_(i,j) ^(k+1) as computed in 1) above is            given for example by FIG. 9. Then, since by the definition,            the lattice Ŷ^(k+1) is a sub-lattice of lattice Y ^(k), then            the optical flow for the Black pixels in lattice Y ^(k) is            already known, see FIG. 10. It therefore remains only to            compute the optical flow for the White pixels in FIG. 10.            However, every “White” pixel in Y ^(k) is surrounded by four            neighboring Black pixels for which the optical flow is            already known. Therefore we can use this information to            predict an optical flow for the surrounded White pixel of            that neighborhood. For example, we can average the            surrounding optical flows to get that value. In FIG. 10, the            optical flow for the “White” pixel denoted by w, is            therefore predicted by the average of the optical flows for            pixels a,b,c and d, yielding the pixel denoted by W.

    -   4) In both the Encoder and Decoder, we now complete the        definition of Y ^(k) as follows. First we note that again since        Ŷ^(k+1) is a sub-lattice of Y ^(k), we already know half of the        pixels of Y ^(k), for example the Black pixels in FIG. 10. Next,        to compute the remaining pixels, we can use spatial and/or        temporal prediction.

    -   5) Spatial prediction: We use the known pixel values of Y ^(k)        to predict the remaining pixel values of Y ^(k). For example, we        may look at a pixel's 4 surrounding neighbors as in diagram 1110        of FIG. 11, or at a pixel's 16 surroundings neighbors as in        diagram 1120 of FIG. 11, etc. Using these surrounding neighbors        we may now predict the White pixel value using averaging,        interpolation, etc.

    -   6) Temporal prediction: We use the computed proposed optical        flows between the lattice Y ^(k) and the respective lattices        {tilde over (Z)}_(i,j) ^(k) of the Quincunx Spatial Pyramid of        Z, computed in 3) above, to predict the missing pixels. For        example, in FIG. 10, we can predict the value of the White pixel        w in Y ^(k) from the corresponding value W in {tilde over        (Z)}_(i,j) ^(k).

    -   7) By now, we have completely defined the lattice Y ^(k) which        should be an approximation to the lattice of {tilde over        (Y)}^(k). In practice, however, not all pixels in Y ^(k) are        good predictions to the corresponding pixels in {tilde over        (Y)}^(k). Therefore, some corrections are needed. Accordingly,        in the Encoder, we may now compute the difference

D ^(k) ={tilde over (Y)} ^(k) −Y ^(k),

-   -   -   and decide to improve the values of some of the pixels in            the so predicted lattice Y ^(k). For example, we can            compress that difference, using some known compression            method, and send the compressed bit stream to the Decoder.

    -   8) In the Decoder and the Encoder, we then decompress the        respective bit stream to obtain {circumflex over (D)}^(k).        Finally, in both the Decoder and the Encoder we add that        compressed difference to compute {tilde over        (Ŷ)}^(k)={circumflex over (D)}^(k)+Y ^(k).

    -   9) in both the Decoder and Encoder we compute Ŷ^(k) by        de-blurring {tilde over (Ŷ)}^(k).

The Compressed Bit Stream

The compressed bit stream consists of the aggregate compressed bitstreams for:

-   -   Ŷ^(L) as computed in 1) of Step L above; and for    -   {circumflex over (D)}^(k), k=L−1, . . . , 0, as computed in 7)        of Step k above.

REFERENCE

Ref[1] https://en.wikipedia.org/wiki/Moving_Picture_Experts_Group.

Ref[2] https://en.wikipedia.org/wiki/Optical_flow.

Ref[3]https://en.wikipedia.org/wiki/Embedded_Zerotrees_of_Wavelet_transforms.

Patents

-   -   Pat [1] Ilan Bar-On and Oleg Kostenko, Multi-Level        Spatial-Temporal Resolution Increase of Video, App. No.        PCT/182014/062524    -   Pat [2] Ilan Bar-On and Oleg Kostenko, Video Compression Method,        App. No. PCT/182013/059007    -   Pat [3] Ilan Bar-On and Oleg Kostenko, A New Universal Video        Codec, App. No. PCT/IB2015/054735.

1. A method of representing image data comprising a spatial pyramid datastructure of order L, said method comprising: providing an initialrepresentation of said image data, said representation comprising a setof lattices of level 0; connecting each said lattices of level 0 to saidimage data; performing L reduction steps each comprising: for each oneof at least one current lattice of the current level, optionallyblurring said current lattice; providing a plurality of down sampledsub-lattices of said current optionally blurred lattice, said at leastone current lattice being the parent lattice of said respectiveplurality of down sampled sub-lattices; and connecting said plurality ofdown sampled sub-lattices to said respective parent lattice, whereinsaid plurality of down sampled sub-lattices of the current level,comprise the set of current lattices for the next level.
 2. The methodof claim 1, wherein said initial representation of the image datacomprises a set of lattices comprising fractional shifts of the imagedata.
 3. The method of claim 2, wherein said set of lattices comprisesonly one lattice.
 4. The method of claim 1, wherein said providing downsampled sub-lattices comprises performing down sampling using theQuincunx down sampling method.
 5. The method of claim 1, wherein said atleast one current lattice comprises all current lattices of the currentlevel.
 6. A method of representing N+1 frames of image data comprising:providing a set of N frames, and using the method of claim 1 to create aspatial pyramid structure of order L for each one of said frames in saidset of N frames; providing another frame and creating a simple spatialpyramid data structure of order L for said another frame as follow: saidanother frame comprising the current lattice of level 0; performing Lreduction steps each comprising: optionally blurring said currentlattice; providing a down sampled sub-lattice of said current optionallyblurred lattice, said current lattice being the parent lattice of saidrespective down sampled sub-lattice; and connecting said down sampledsub-lattice to said respective parent lattice, wherein said down sampledsub-lattice of the current level, comprises the current lattice for thenext level.
 7. The method of claim 6, used for performing videocompression, wherein said set of N frames comprises a set of N alreadydecoded frames and wherein said another frame comprises a current frameto be encoded and decoded, comprising: compressing /decompressing thelowest level lattice of said simple spatial is pyramid using said Nspatial pyramids, said decompressed lattice being the currentdecompressed lattice: performing L raise steps each comprising:computing the optical flows between said current decompressed latticeand the respective lattice of each of said N spatial pyramids; upscaling said optical flows to optical flows for the next higher level;predicting a decompressed lattice of the next higher level using said upscaled optical flows and the respective lattices of said N spatialpyramids: computing an enhancement data for said decompressed latticeusing the respective lattice of said simple spatial pyramid;compressing/decompressing said enhancement data; adding saiddecompressed enhancement data to said decompressed lattice of the nexthigher level to create an enhanced decompressed lattice of the nexthigher level; and optionally deblurring said enhanced decompressedlattice of the next higher level to create the current decompressedlattice.
 8. The method of claim 7 where said predicting the decompressedlattice of the next higher level comprises spatial and/or temporalprediction.
 9. The method of claim 7, wherein said N already decodedframes comprise N already decoded frames of an original video, andwherein said current frame to be encoded and decoded comprises a newframe of the original video, wherein said creating the N spatialpyramids for said N frames, is performed in both the Encoder andDecoder; wherein said creating the simple spatial pyramid for saidanother frame is performed in the Encoder; wherein said compressing thelowest level lattice of said simple spatial pyramid is performed in theEncoder; said decompressing said compressed lowest level lattice isperformed in both the Encoder and Decoder; wherein said performing Lraise steps is performed in the Encoder/Decoder as follows: saidcomputing the optical flows is performed in both the Encoder andDecoder; said up scaling of said optical flows is performed in both theEncoder and Decoder; said predicting the decompressed lattice of thenext higher level is performed in both the Encoder and Decoder; saidcomputing enhancement and said compressing of said enhancement data isperformed in the Encoder; said decompressing of said compressedenhancement data is performed in both the Encoder and Decoder; and saidadding said decompressed enhancement data and said optionally deblurringsaid enhanced decompressed lattice is performed in both the Encoder andDecoder.
 10. The method of claim 9 where said predicting thedecompressed lattice of the next higher level comprises spatial and/ortemporal prediction.